i-Code: An Integrative and Composable Multimodal Learning Framework

Human intelligence is multimodal; we integrate visual, linguistic, and acoustic signals to maintain a holistic worldview. Most current pretraining methods, however, are limited one or two modalities. We present i-Code, self-supervised framework where users may flexibly combine the modalities of vision, speech, language into unified general-purpose vector representations. In this framework, data from each modality first given pretrained single-modality encoders. The encoder outputs then integrated with multimodal fusion network, which uses novel merge- co-attention mechanisms effectively information different entire system end-to-end new objectives including masked unit modeling cross-modality contrastive learning. Unlike previous research using only video for pretraining, i-Code can dynamically process single, dual, triple-modality during training inference, projecting combinations single representation space. Experimental results demonstrate how outperform state-of-the-art techniques on five understanding tasks benchmarks, improving by as much 11% demonstrating power integrative pretraining.